CUDA: add DGX Spark network expert/tensor parallelism - #754

Summary

• Add guarded two- and four-rank NCCL expert parallelism and tensor parallelism for supported DeepSeek V4 Flash 0731 and GLM 5.2 GGUF layouts on CUDA.
• Keep every rank in one mirrored session graph while partitioning routed experts and model-specific decode work. Complete heads, FFN lanes, and output rows are sliced so the supported default path preserves the reference arithmetic; dense prefill/shared work remains replicated where the optimized batch kernels are shape-sensitive.
• Wire the same leader/worker lifecycle through ds4, ds4-bench, ds4-server, ds4-agent, ds4-eval, and the official-continuation scorer.
• Add topology/preflight validation, guarded unified-memory planning, multi-rank control protocol coverage, CUDA row-slice/kernel tests, documentation, and release QA gates.

Related upstream issues

• Support for 2 GX10s cluster with 128GB each - support tensor parallelism #262: implements the requested two-node, 128 GiB-per-node CUDA tensor-parallel cluster path.
• ds4-server: expose tensor-parallel roles (field report: 2x M4 Max TB5, TP 22.7 t/s vs server-pipeline ~20) #651: exposes network tensor-parallel roles through ds4-server (and also wires the agent, eval, benchmark, and scorer frontends).
• GLM-5.2 (glm-dsa) prefill fails on CUDA / DGX Spark GB10 (sm_121): "cuda GLM prefill failed" with --ssd-streaming #595: addresses the question about CUDA cross-node TP for GLM 5.2; the issue's single-node CUDA --ssd-streaming prefill failure remains outside this PR.
• Higher inference throughput on DGX possible? #469: provides the cross-DGX TP/EP mechanism discussed there; continuous batching and vLLM-style aggregate throughput remain outside scope.
• Multi-GPU for nvidia #123: overlaps the networked four-Spark use case discussed in the thread; the separate single-host multi-GPU path is unchanged.

Supported paths and safety

• CUDA network transport is NCCL (--transport auto or nccl) with rank worlds of 2 or 4.
• The tested GLM path uses ownership-aware IQ2_XXS routed gate/up tensors, IQ2_XXS or Q2_K routed down projections, and Q8_0 dense/attention projections for full TP.
• The tested DeepSeek path is the 0731 Q2 tensor mix with IQ2_XXS routed gate/up, Q2_K routed down, and Q8_0 attention/shared/output projections.
• Every host still needs the complete GGUF on local disk; only resident model work is sharded.
• A DGX Spark unified-memory guard retains 32 GiB headroom. The 196.6 GiB GLM fixture is deliberately refused on two 128 GiB Sparks and tested across four.
• External MTP/DSpark drafting, GLM MTP, directional steering, SSD streaming, unsupported routed layouts, and pipeline layer roles are rejected before model load in network mode.
• DS4_CUDA_TP_FAST_ALIGNED_EXPERTS=1 remains an explicit DeepSeek throughput opt-in because its aligned representation changes rounding; it is not the default parity path.

Validation completed

• Warning-free CUDA Spark builds of all five frontends independently on 192.168.2.30 through .33; reviewed sources are byte-identical across ranks.
• Clean CPU build of all five frontends.
• Local 2/4-rank EP and TP control protocol, mixed-batch frames, worker ACK/failure propagation, and failed preflight path.
• Shared CLI/frontend/scorer parser suite: 101/101 checks passed on Linux and
  93/93 backend-adjusted checks passed on macOS.
• CUDA sampling, cross-device/kernel regression, GLM attention/indexer/decode, Q8 TP row slicing, long-context top-k, and aligned Q8 output-row tests passed on GB10.
• Extractor, agent, layer-pack (97/97), multi-GPU placement (101/101), GPU argument, Q4_K, and MXFP4 unit gates passed.
• ds4_test --server passes.
• GLM GGUF SHA-256 matched on all four Sparks (a49de64c...b07fea0).
• Four-rank GLM model-backed frontend smokes:
  • TP ds4-server: real /v1/chat/completions request completed; graceful shutdown stopped all ranks with status 0.
  • EP ds4-agent: the non-interactive prompt returned exactly OK; all ranks exited cleanly.
  • TP ds4-eval: one 64-token GPQA case completed and all ranks exited cleanly (the quantized model selected C for a fixture whose answer is B, so the quality harness itself returned 1).
• DeepSeek exact EP and TP official-continuation scores agree (avg_nll=0.403166544, 61/100 greedy first-token matches); the matched single-Spark reference measured avg_nll=0.406647149.
• GLM four-rank EP and TP 32-token greedy/logprob traces are byte-identical without ordered reduction.
• GLM four-rank TP completed the tracked 100-case official-continuation fixture on this commit: 2,299 local target tokens, avg_nll=0.468965305, 83/100 greedy first-token matches, avg_lcp=6.850, and 99.768% API top-logprob mapping coverage. All workers exited with status 0.
• Apple M3 Max (128 GiB, macOS 26.5.1) Metal gate:
  • Warning-free clean build of ds4, ds4-server, ds4-bench, ds4-eval, and ds4-agent.
  • Full make test passed against DeepSeek-V4-Flash-0731-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-ds4.gguf; Metal mapped 82,697.67 MiB and passed the 30,474-token long-context, tool-call, official logprob, local-golden, kernel, and tensor-equivalence checks.
  • make test-metal-session-batch passed with 2 sessions, 6 decode steps, an 8-token mixed suffix, and exact logits.

The network control protocol is intended for trusted cluster networks and is not authenticated or release-stable yet; every rank should run the same commit and model bytes.